Apparatuses and methods for three-dimensional dental segmentation using dental image data

ABSTRACT

Methods and apparatuses (including systems and devices), including computer-implemented methods for segmenting, correcting and/or modifying a three-dimensional (3D) model of a subject&#39;s oral cavity to determine individual components such as teeth, gingiva, tongue, palate, etc., that may be selective and/or collectively digitally manipulated. In some implementations, artificial intelligence uses libraries of labeled 2D images and 3D dental models to learn how to segment a 3D dental model of a subject&#39;s oral cavity using 2D images, height map and/or other data and projection values that relate the 2D images to the 3D model. As noted herein, the dental classes can include a variety of intra-oral and extra-oral objects and can be represented as binary values, discrete values, a continuum of height map data, etc. In some implementations, several dental classes are predicted concurrently.

PRIORITY CLAIM

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/896,509, filed Sep. 5, 2019, titled “APPARATUSES AND METHODS FORTHREE-DIMENSIONAL DENTAL SEGMENTATION,” which is herein incorporated byreference in its entirety.

INCORPORATION BY REFERENCE

All publications and patent applications mentioned in this specificationare herein incorporated by reference in their entirety to the sameextent as if each individual publication or patent application wasspecifically and individually indicated to be incorporated by reference.

FIELD

The methods and apparatuses described herein relate to computer-assisteddentistry and orthodontics, and more particularly to processing ofthree-dimensional (3D) dental models using data from dental scans and/ordental images.

BACKGROUND

Tools for two-dimensional (2D) and three-dimensional (3D) digital imagetechnology are becoming increasingly useful in assisting in dental andorthodontic treatment. Treatment providers may use some form of digitalimage technology to study the dentitions of subjects. 2D and 3D imagedata may be used to form a digital model of a subject's dentition,including models of individual dentition components. Such models areuseful, among other things, in developing an orthodontic treatment planfor the subject, as well as in creating one or more orthodonticappliances to implement the treatment plan. While it would be desirableto accurately segment, modify, update, and/or process 3D dental models,existing techniques make it difficult to do so.

SUMMARY OF THE DISCLOSURE

Described herein are systems and methods for generating high accurateand segmented models of a subject's oral cavity. These models may beeasily manipulated by a dental practitioner, e.g., use such as a doctor,orthodontist, dental professional, etc. These methods and apparatusesmay include image generation, generative Adversarial Networks, and thelike.

For example, described herein are computer-implemented methods. Thesemethods may be configured to segment a 3D model by processing aplurality of carefully selected 2D images, including segmenting these 2Dimages, combining the segmentation data, and applying the 2Dsegmentation data to a 3D model. For example a method may include:identifying a plurality of two-dimensional (2D) images of a subject'soral cavity, wherein the 2D images correspond to a digitalthree-dimensional (3D) model of the subject's oral cavity; processingthe plurality of 2D images to segment each 2D image into a plurality ofdifferent structures; and projecting the segmented 2D images onto the 3Dmodel to form a segmented 3D model.

The method may also include collecting the plurality of 2D images. Forexample, collecting the plurality of 2D images by identifying a view ofthe 3D model and generating a 2D projection of the 3D model from theview, and/or collecting the plurality of 2D images from scanned imagesof the subject's oral cavity.

Any of these methods may also or alternatively include modifying the 2Dimages; for example, adjusting a height map of each 2D image. Processingthe plurality of 2D images may include applying a trainedmachine-learning agent to segment each of the 2D images. For example,processing may comprise using a conditional Generative AdversarialNetwork.

In any of these methods, projecting the segmented 2D images onto the 3Dmodel may include resolving conflicts between the segmentation of each2D image where a plurality of 2D images project to the same location onthe 3D model. For example, resolving the conflicts may comprise applyingBayes' Theorem or voting on a per-location basis in the 3D model basedon the plurality of 2D images that project onto the location.

A computer-implemented method may include: generating a plurality ofinterproximal separation planes between teeth of a digitalthree-dimensional (3D) model of a subject's oral cavity; collecting atwo-dimensional (2D) images corresponding to each of one or more of:buccal, lingual and occlusal views, wherein the 2D images correspond toprojections of the 3D model that are taken perpendicular to aninterproximal separation plane of the plurality of interproximalseparation planes; segmenting the 2D images to identify the boundariesbetween different components within the 2D images, wherein thecomponents comprise teeth; combining the segmented 2D images to form aconsensus segmentation of locations on the 3D model; and applying theconsensus segmentation to the 3D model to form a segmented 3D model ofthe subject's oral cavity.

Any of these methods may include numbering the teeth of the 2D imagesusing the 3D model or numbering the teeth in the 2D images and applyingthe numbering to the 3D model.

Any of these methods may include enhancing the 2D images prior tosegmenting the 2D images in order to generate an enhanced 3D model. Forexample, enhancing may include adjusting the interproximal regionbetween two or more teeth in the 2D images. Segmenting may includeapplying a trained machine-learning agent to segment each of the 2Dimages. In some variations, segmenting comprises using a conditionalGenerative Adversarial Network. Any of these methods may includesegmenting the gingiva by identifying the segmented teeth in thesegmented 3D model.

As mentioned, combining the segmented 2D images may include applyingBayes' Theorem or voting for specific locations on the 3D model that arerepresented by a plurality of 2D images.

In general, any the methods described herein may be performed by asystem including one or more processors and a memory includinginstructions to perform the method. For example, as described herein, asystem may include: one or more processors; a memory coupled to the oneor more processors, the memory configured to store computer-programinstructions, that, when executed by the one or more processors, performa computer-implemented method comprising: identifying a plurality oftwo-dimensional (2D) images of a subject's oral cavity, wherein the 2Dimages correspond to a digital three-dimensional (3D) model of thesubject's oral cavity; processing the plurality of 2D images to segmenteach 2D image into a plurality of different structures; and projectingthe segmented 2D images onto the 3D model to form a segmented 3D model.The system may be configured to perform any of the steps describedherein. These systems may include any of the modules or enginesdiscussed herein.

For example, a system may include: one or more processors; a memorycoupled to the one or more processors, the memory configured to storecomputer-program instructions, that, when executed by the one or moreprocessors, perform a computer-implemented method comprising: generatinga plurality of interproximal separation planes between teeth of adigital three-dimensional (3D) model of a subject's oral cavity;collecting a two-dimensional (2D) images corresponding to each of one ormore of: buccal, lingual and occlusal views, wherein the 2D imagescorrespond to projections of the 3D model that are taken perpendicularto an interproximal separation plane of the plurality of interproximalseparation planes; segmenting the 2D images to identify the boundariesbetween different components within the 2D images, wherein thecomponents comprise teeth; combining the segmented 2D images to form aconsensus segmentation; and applying the consensus segmentation to the3D model to form a segmented 3D model of the subject's oral cavity.

For example, described herein are computer-implemented methods, any ofwhich may include: identifying a plurality of two-dimensional (2D)images of a subject's oral cavity, wherein the 2D images correspond to adigital three-dimensional (3D) model of the subject's oral cavity;processing the plurality of 2D images to segment each 2D image into aplurality of different structures; and projecting the segmented 2Dimages onto the 3D model to form a segmented 3D model.

The methods and apparatuses described herein may also include collectingthe plurality of 2D images. For example, any of these methods orapparatuses may include collecting the plurality of 2D images byidentifying a view of the 3D model and generating a 2D projection of the3D model from the view. In some variations, the method, or an apparatusconfigured to perform the method, may include collecting the pluralityof 2D images from scanned images of the subject's oral cavity.

In particular, any of these methods and apparatuses described herein maybe configured to include modifying the 2D images. For example, modifyingmay include adjusting a height map of each 2D image.

In general, processing the plurality of 2D images may include applying atrained machine-learning agent to segment each of the 2D images. Forexample, processing may include using a conditional GenerativeAdversarial Network.

Any of these methods (or an apparatus configured to perform them) mayinclude projecting the segmented 2D images onto the 3D model whichcomprises resolving conflicts between the segmentation of each 2D imagebased on the projection onto the 3D model.

In general, resolving the conflicts comprises applying Bayes' Theorem orvoting. For example, combining the segmented 2D images may compriseapplying Bayes' Theorem for a plurality of 2D images which representoverlapping locations on the 3D model in order to create a probabilitydistribution of dental types per location in the 3D model. Any of thesemethods may include numbering the teeth of the 2D images using the 3Dmodel or numbering the teeth in the 2D images in order to find the toothnumbers of locations on the 3D model.

For example, segmenting comprises applying a trained machine-learningagent to segment each of the 2D images into their relevant dental types.Segmenting may include using a conditional Generative AdversarialNetwork. Segmenting may include segmenting the gingiva by identifyingthe segmented teeth and gingiva in the segmented 3D model.

A computer-implemented method may include: generating a plurality ofinterproximal separation planes between teeth of a digitalthree-dimensional (3D) model of a subject's oral cavity; collecting atwo-dimensional (2D) images corresponding to each of one or more of:buccal, lingual and occlusal views, wherein the 2D images correspond toprojections of the 3D model that are taken perpendicular to aninterproximal separation plane of the plurality of interproximalseparation planes; segmenting the 2D images to identify the boundariesbetween different components within the 2D images, wherein thecomponents comprise teeth; combining the segmented 2D images to form aconsensus segmentation; and applying the consensus segmentation to the3D model to form a segmented 3D model of the subject's oral cavity.

As mentioned, also described herein are systems, including systemsconfigured to perform any of the methods described herein. For example,a system may include: one or more processors; a memory coupled to theone or more processors, the memory configured to store computer-programinstructions, that, when executed by the one or more processors, performa computer-implemented method comprising: identifying a plurality oftwo-dimensional (2D) images of a subject's oral cavity, wherein the 2Dimages correspond to a digital three-dimensional (3D) model of thesubject's oral cavity; processing the plurality of 2D images to segmenteach 2D image into a plurality of different structures; and projectingthe segmented 2D images onto the 3D model to form a segmented 3D model.

A system may include: one or more processors; a memory coupled to theone or more processors, the memory configured to store computer-programinstructions, that, when executed by the one or more processors, performa computer-implemented method comprising: generating a plurality ofinterproximal separation planes between teeth of a digitalthree-dimensional (3D) model of a subject's oral cavity; collecting atwo-dimensional (2D) images corresponding to each of one or more of:buccal, lingual and occlusal views, wherein the 2D images correspond toprojections of the 3D model that are taken perpendicular to aninterproximal separation plane of the plurality of interproximalseparation planes; segmenting the 2D images to identify the boundariesbetween different components within the 2D images, wherein thecomponents comprise teeth; combining the segmented 2D images to form aconsensus segmentation; and applying the consensus segmentation to the3D model to form a segmented 3D model of the subject's oral cavity.

The computer-implemented method incorporated as part of the system mayinclude any of the steps and variations of steps described above andherein.

Also described herein are methods, including methods of segmenting a 3Dmodel, that include: accessing a plurality of first two-dimensional (2D)images, wherein the plurality of first 2D images: represents a subject'soral cavity, each has first areas that can be segmented into a pluralityof dental classes, each has a first relationship to a firstthree-dimensional (3D) model of the subject's oral cavity, and each hasfirst height map data representing distances between the subject's oralcavity and an image capture device; accessing one or more automatedmachine learning agents trained to modify one or more second 3D modelsinto the plurality of dental classes, the trained modifications usingsecond height map data of a plurality of second 2D images and furtherusing second relationships between the plurality of second 2D images andthe one or more second 3D models; instructing the one or more automatedmachine learning agents to use the first height map data to modify thefirst areas of the plurality of first 2D images to get a plurality ofmodified first 2D images; and using the first relationships and theplurality of modified first 2D images to modify first mesh regions ofthe first 3D model corresponding to the first areas of the plurality offirst 2D images.

Any of these methods may also include: gathering the plurality of second2D images from a training datastore; identifying one or moremodifications to second areas of the plurality of second 2D images;training the one or more automated machine learning to use the secondheight map data to provide the one or more modifications to the secondareas of the plurality of second 2D images to get a plurality ofmodified second 2D images; and training the one or more automatedmachine learning to use the second relationships and the plurality ofmodified second 2D images to modify second mesh regions of the one ormore second 3D models corresponding to the second areas.

In general, accessing the plurality of first 2D images may includegathering the plurality of first 2D images. The first relationship maybe represented by 2D-3D projection values to project portions of theplurality of first 2D images onto the first 3D model.

In any of these methods, the one or more automated machine learningagents may include a classifier trained to modify the one or more second3D models. For example, the one or more automated machine learningagents may comprise a Generative Adversarial Network (GAN) trained tomodify the one or more second 3D models. In some variations, the one ormore automated machine learning agents comprises a conditionalGenerative Adversarial Network (cGAN) trained to segment the one or moresecond 3D models into the plurality of dental classes.

The first 3D model may include a 3D mesh of the subject's oral cavity,the one or more second 3D models comprise one or more 3D meshes of aplurality of oral cavities, or some combination thereof.

In any of these methods, the step of using the first relationships andthe modified first 2D images to modify the first 3D model may includemapping one or more pixel values from pixels of the plurality ofmodified first 2D images onto one or more faces of a mesh of the first3D model. For example, using the first relationships and the modifiedfirst 2D images to modify the first 3D model may include representingthe plurality of dental classes using a plurality of color channels.

The methods described herein may include instructing the one or moreautomated machine learning agents to use a plurality of data types fromthe plurality of first 2D images modify the first areas of the pluralityof first 2D images to get the plurality of modified first 2D images. Forexample, these methods may include instructing the one or more automatedmachine learning agents to use color data, count map data, texture data,grading data, or some combination thereof, from the plurality of first2D images modify the first areas of the plurality of first 2D images toget the plurality of modified first 2D images.

In any of these methods, using the first relationships and the pluralityof modified first 2D images to modify the first mesh regions may includesegmenting the first 3D model using the modified first 2D images and thefirst relationships. For example, the trained modifications may compriseone or more segmentations segmenting the second 3D models into aplurality of dental classes. In some variations the trainedmodifications comprise one or more segmentations segmenting the second3D models into a plurality of dental classes. At least some of theplurality of dental classes may comprise teeth, gums, and excessmaterials, or some combination thereof. In some variations the trainedmodifications may include one or more segmentations segmenting thesecond 3D models into a plurality of dental classes. At least some ofthe plurality of dental classes may comprise a plurality of anatomicaltooth identifiers.

In any of these methods, the trained modifications may include one ormore segmentations segmenting the second 3D models into a plurality ofdental classes, and at least some of the plurality of dental classes maycomprise extra-oral objects, dental appliances, oral soft tissue, orsome combination thereof. For example, the trained modifications mayinclude one or more segmentations segmenting the second 3D models into aplurality of dental classes, and the plurality of dental classes maycomprise binary values, discrete values, or some combination thereofrepresenting existence or non-existence of one or more portions ofdental anatomy. In some variations the trained modifications compriseone or more segmentations segmenting the second 3D models into aplurality of dental classes, and the plurality of dental classes maycomprise continuous values related a target height map for the first 3Dmodel. The trained modifications may comprise one or more segmentationssegmenting the second 3D models into a plurality of dental classes, andthe first relationships may represent projections of pixels on the eachof the plurality of first 2D images to one or more faces of a mesh ofthe first 3D model.

The first relationships and the plurality of modified first 2D images tomodify the first mesh regions may include improving representations ofone or more features of the first 3D model using the modified first 2Dimages and the first relationships.

Any of these methods may also include: gathering the first 3D modeland/or generating the first plurality of 2D images using one or more3D-2D projection values to transfer portions of the first 3D model ontoportions of the plurality of first 2D images.

The first areas of the plurality of first 2D images may comprise regionsof limited or missing height map data. In some variations, instructingthe one or more automated machine learning agents to use the firstheight map data to modify the first areas of the plurality of first 2Dimages may include adding new height map data to the first areas. Forexample, the plurality of first 2D images may comprise a plurality of 2Dperspectives of the subject's oral cavity; in some variations the firstareas of the plurality of first 2D images comprises an oral component tobe modified. The method may further include instructing the one or moreautomated machine learning agents to use the first height map data tomodify the first areas of the plurality of first 2D images comprisesresolving one or more conflicts between conflicting representations ofthe oral component.

The plurality of first 2D images may comprise a plurality of 2Dperspectives of the subject's oral cavity. The first areas of theplurality of first 2D images may include an oral component to bemodified. Instructing the one or more automated machine learning agentsto use the first height map data to modify the first areas of theplurality of first 2D images may include resolving one or more conflictsbetween conflicting representations of the oral component by using astatistical process to reconcile the one or more conflicts. For example,the first areas of the plurality of first 2D images may compriseinterproximal regions of teeth within the subject's oral cavity, andinstructing the one or more automated machine learning agents to use thefirst height map data to modify the first areas of the plurality offirst 2D images may include updating representations of theinterproximal regions. The plurality of first 2D images may comprisebuccal views of the subject oral cavity, lingual views of the subjectoral cavity, occlusal views of the subject oral cavity, or somecombination thereof.

In some variations, the first relationships may provide mesial-distalprojections of the first 3D model onto the plurality of first 2D images.For example, the first relationship may be represented by: 3D-2Dprojection values to transfer portions of the first 3D model onto theplurality of first 2D images, Delaunay triangulation, marching cubes, orsome combination thereof.

Also described herein are systems including: one or more processors;memory storing computer-program instructions that, when executed by theone or more processors cause the system to implement a methodcomprising: accessing a plurality of first two-dimensional (2D) images,wherein the plurality of first 2D images: represents a subject's oralcavity, each has first areas that can be segmented into a plurality ofdental classes, each has a first relationship to a firstthree-dimensional (3D) model of the subject's oral cavity, and each hasfirst height map data representing distances between the subject's oralcavity and an image capture device; accessing one or more automatedmachine learning agents trained to modify to one or more second 3Dmodels into the plurality of dental classes, the trained modificationsusing second height map data of a plurality of second 2D images andfurther using second relationships between the plurality of second 2Dimages and the one or more second 3D models; instructing the one or moreautomated machine learning agents to use the first height map data tomodify the first areas of the plurality of first 2D images to get aplurality of modified first 2D images; and using the first relationshipsbetween the plurality of first 2D images and the first 3D models, andusing the plurality of modified first 2D images to modify first meshregions of the first 3D model corresponding to the first areas of theplurality of first 2D images.

Also described herein are methods including: gathering a plurality offirst two-dimensional (2D) images, wherein the plurality of first 2Dimages: represents a subject's oral cavity, each has first areas thatcan be segmented into a plurality of dental classes, each has firstprojection values in relation to a first three-dimensional (3D) model ofthe subject's oral cavity, and each has first height map datarepresenting distances between the subject's oral cavity and an imagecapture device; accessing one or more automated machine learning agentstrained to segment one or more second 3D models into the plurality ofdental classes, the trained segmenting using second height map data of aplurality of second 2D images and further using second projection valuesrelating the plurality of second 2D images to the one or more second 3Dmodels; instructing the one or more automated machine learning agents touse the first height map data to segment the first areas of theplurality of first 2D images into the plurality of dental classes to geta plurality of segmented first 2D images; and using the first projectionvalues and the plurality of segmented first 2D images to segment thefirst 3D model of the subject's oral cavity into the plurality of dentalclasses.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe claims that follow. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIGS. 1A-1D illustrate examples of intraoral scans having regions thatmay be difficult to resolve using traditional segmentation techniques.

FIG. 2A shows one variation of a schematic illustration of a system forthree-dimensional (3D) modeling of a subject's oral cavity.

FIG. 2B schematically illustrates one example of a 2D image processingengine that may be part of a system for 3D modeling of a subject's oralcavity.

FIG. 2C schematically illustrates one example of a componentconstruction engine portion of a system for 3D modeling of a subject'soral cavity.

FIG. 2D illustrates one example of a method of segmenting a 3D model ofa subject's oral cavity to identify component parts, such as individualteeth, gingiva, etc.

FIGS. 3A-3C illustrate training an agent, e.g., a machine learningagent, to recognize individual teeth from images of a subject's teeth.FIG. 3A illustrates mapping of height map inputs to manually identifiedsegmented images (FIG. 3B), and using this information to predict labelsfrom 2D height maps (FIG. 3C).

FIG. 4A is an example of a 2D projection of an occlusive view of a raw3D model.

FIG. 4B shows the example projection of FIG. 4A after analyzing andlabeling the 2D image of FIG. 4A (and others) and applying this analysisand labeling to a 3D model of the subject's dentition; in FIG. 4B justthe segmented teeth are shown.

FIGS. 5A-5F illustrate information from 2D images that may be processedby the methods and systems described herein, and used to segment and/ormodify the 3D models as described herein. FIG. 5A illustrates height mapinformation, FIG. 5B illustrates a count map, FIG. 5C is a grades map,FIG. 5D illustrates texture, FIG. 5E shows labeled regions (e.g., emptyspace of tooth space), and FIG. 5F shows a prediction of labeled regionsbased on a machine learning agent as described below.

FIGS. 6A-6B illustrate segmentation prediction based on a height map(shown in FIG. 6A), which may be used to differentiate between tooth(shown in white) and non-tooth (e.g., gingiva and excess materials inblack and gray, respectively).

FIG. 7 illustrates one example of a method of forming a segmented 3Dmodel of a subject's oral cavity that includes separately andspecifically identified component parts (e.g., individual teeth,gingiva, etc.).

FIGS. 8A-8B schematically illustrate a putative numbering based on aheight map.

FIG. 8A shows an unmodified buccal view of a subject's lower arch. FIG.8B shows an example of a labeled set of teeth from FIG. 8A. In thisexample individual teeth are marked by both color and by number (e.g.,alphanumeric label).

FIG. 9 schematically illustrates one example of tooth numbering based onheight map information for one, or more preferably more, 2D images.

FIGS. 10A-10C illustrate the steps of FIG. 9, showing a height map (FIG.10A), ordering the teeth in the image (FIG. 10B), and determining aprobably tooth numbering order from the height map information (FIG.10C).

FIGS. 11A-11B illustrate one variation in which interproximal spacing isidentified in a plurality of 2D slices or images of the subject's teeth,and (as shown in FIG. 11B) planes showing most likely interproximalseparation between adjacent teeth.

FIGS. 12A-12C show one variation of a method of segmenting a subject'steeth in which 2D projections and/or captured images are identified,such as projections from buccal (FIG. 12A), lingual (FIG. 12B) andocclusal (FIG. 12C) views taken from the original (unsegmented andunmodified) 3D model.

FIGS. 13A-13B illustrates a step of enhancing the identified projectionsand/or captured 2D images as described herein. FIG. 13A (similar to FIG.12A) shows a buccal projection of the 3D model and FIG. 13B shows anenhanced version of the projection.

FIGS. 14A-14B illustrates further processing of 2D images, showing theuse of the enhanced 2D images to determine segmentation.

FIG. 15A-15F illustrate identifying different 2D views of a samecomponent (e.g., tooth) across multiple 2D views, projections orcaptured images.

FIG. 16 illustrate a combined or merged image based on the multipledifferent 2D views such as shown in FIGS. 15A-15F.

FIGS. 17A-17B illustrate modeling of the gingiva following segmentationof the teeth. FIG. 17A shows the mapping of the segmented 2D images tocorresponding mesh points in the original scanned model. FIG. 17B showsthe original model with the identified (e.g., segmented) teeth removed,after a hole-filling technique has been used on the remaining, gingival,tissue.

FIG. 18 shows a completely segmented and/or corrected 3D model formed asdescribed herein.

FIG. 19 shows a treatment planning ecosystem, in accordance with someimplementations.

FIG. 20 illustrates one method of segmenting a 3D model as describedherein.

DETAILED DESCRIPTION

While desirable, accurately segmenting and representing the portions ofa three-dimensional (3D) model of an oral cavity has proven difficult.One issue is that 3D models based on two-dimensional (2D) images,including those with height map information, do not accurately representa subject's oral cavity. Representations of interproximal regions,gingival lines, and other regions may be inaccurate due to operation ofthe hardware and/or software used to capture 2D images of dentition.FIGS. 1A-1D illustrate examples of 3D models 100 a-100 d having regionsthat may be difficult to resolve using traditional segmentationtechniques. In FIG. 1A, for example, a 3D model 100 a comprises a closedregion 105 in an interproximal area between two teeth. In FIG. 1B, a 3Dmodel 100 b comprises a closed region 107 in an interproximal areabetween adjacent teeth. In the examples of FIGS. 1A and 1B, the adjacentteeth around the closed regions 105 and 107 appear unseparated, withinaccurately sized and shaped gaps between the teeth. The 3D model 100 cof FIG. 1C comprises two unclear separation areas 109 between a premolartooth and its neighboring teeth. The 3D model 100 d of FIG. 1D comprisesan unclear separation area 111 between teeth and a gingival line. Theunclear separation areas 109 and 111 do not accurately represent asubject's oral anatomy. For instance, the unclear separation area 109does not show gaps between the premolar and its neighbors. The unclearseparation area 111 does not show the line between the subject's gingivaand teeth. It would be difficult to use the 3D dental models 100 a-100 dfor treatment planning due to inaccurate anatomical representations andother issues. Existing solutions to segment the 3D dental models 100a-100 d involve manually review/labeling or processing the 3D model.Existing solutions are often computationally intensive and/or yieldinaccurate results. The techniques described herein address issuesassociated with existing 3D modeling of a subject's oral cavity.

FIG. 19 shows a treatment planning ecosystem 1900, in accordance withsome implementations. The treatment planning ecosystem 1900 includes acomputer-readable medium 1902, a scanner/camera 1904, a treatmentprofessional system 1906, and a treatment planning system 1908. Thescanner/camera 1904, the treatment professional system 1906, and thetreatment planning system 1908 may be coupled to one another over thecomputer-readable medium 1902. The computer-readable medium 1902represents any transitory or non-transitory computer-readable medium orarchitecture capable of facilitating communication or data transfer. Inone example, the computer-readable medium 1902 may facilitatecommunication between scanner/camera 1904, the treatment professionalsystem 1906, and the treatment planning system 1910. In someimplementations, computer-readable medium 1902 comprises a computernetwork that facilitates communication or data transfer using wirelessand/or wired connections. Examples of computer-readable medium 1902include, without limitation, an intranet, a Wide Area Network (WAN), aLocal Area Network (LAN), a Personal Area Network (PAN), the Internet,Power Line Communications (PLC), a cellular network (e.g., a GlobalSystem for Mobile Communications (GSM) network), portions of one or moreof the same, variations or combinations of one or more of the same,and/or any other suitable network. Computer-readable medium 1902 mayalso comprise a connection between elements inside a single computingdevice (e.g., the scanner/camera 1904, the treatment professional system1906, the treatment planning system 1908, etc.). It is noted that whileFIG. 19 shows the elements as distinct blocks, in variousimplementations, two or more of these blocks can be on the samecomputing device.

The scanner/camera 1904 may include a digital device operative tocapture images. The scanner/camera 1904 may comprise an intraoralscanner, a camera, a desktop/laptop computer system, a mobile phone, akiosk, or some combination thereof. In some implementations, thescanner/camera 1904 captures 2D images of an area of interest along withheight map data, e.g., data representing a distance between a part ofthe scanner/camera 1904 and an object within the area of interest. Inthe dental context, the scanner/camera 1904 may capture a series ofimages of an oral cavity. Each image may have associated with it heightmap data that represents the distance between parts of the oral cavityand the scanner/camera 1904. Height map data may be represented in anyformat. In some implementations, height map data may be represented ascolors, intensities, brightness, or other attributes of pixels on a 2Dimage. The scanner/camera 1904 may also capture projection values foreach 2D image. The projection values may be associated with rotationsand/or translations in space that represent how a 2D image is stitchedinto a 3D representation of the area of interest. As noted herein,projection values may represent how pixels on 2D images are projected toa face of a mesh on a 3D model of the area of interest. A “3D model” ofa subject's dentition, as used herein, may include a three-dimensionalrepresentation of one or more surfaces corresponding to physicalcontours of the subject's dentition. A 3D model may include a set ofshapes (e.g., triangles), that when combined together, form a “mesh” orcontours of the 3D model. Each shape may comprise a “face” of the 3Dmodel.

In some implementations, the scanner/camera 1904 captures data aboutcolor inputs and/or data that represents the texture of surfaces withinan area of interest. The scanner/camera 1904 may record scan quality,e.g., data representing whether 2D images accurately represent an areaof interest and/or whether there are flaws, such as holes or unclearareas, within 2D images of an area of interest. In some implementations,the scanner/camera 1904 captures data related to numbers of raw scanscontributing to scan pixels within 2D images.

The treatment professional system 1906 may include a computing devicecapable of reading computer-executable instructions. The treatmentprofessional system 1906 may be, for example, a desktop computer, atablet computing device, a laptop, a smartphone, an augmented realitydevice, or other consumer device. Additional examples of the treatmentprofessional system 1906 include, without limitation, laptops, tablets,desktops, servers, cellular phones, Personal Digital Assistants (PDAs),multimedia players, embedded systems, wearable devices (e.g., smartwatches, smart glasses, etc.), smart vehicles, smart packaging (e.g.,active or intelligent packaging), gaming consoles, Internet-of-Thingsdevices (e.g., smart appliances, etc.), variations or combinations ofone or more of the same, and/or any other suitable computing device.

In various implementations, the treatment professional system 1906 isconfigured to interface with a dental professional. A “dentalprofessional” (used interchangeably with dentist, orthodontist, anddoctor herein) as used herein, may include any person with specializedtraining in the field of dentistry, and may include, without limitation,general practice dentists, orthodontists, dental technicians, dentalhygienists, etc. A dental professional may include a person who canassess, diagnose, and/or treat a dental condition. “Assessment” of adental condition, as used herein, may include an estimation of theexistence of a dental condition. An assessment of a dental conditionneed not be a clinical diagnosis of the dental condition. In someembodiments, an “assessment” of a dental condition may include an “imagebased assessment,” that is an assessment of a dental condition based inpart or on whole on photos and/or images (e.g., images that are not usedto stitch a mesh or form the basis of a clinical scan) taken of thedental condition. A “diagnosis” of a dental condition, as used herein,may include a clinical identification of the nature of an illness orother problem by examination of the symptoms. “Treatment” of a dentalcondition, as used herein, may include prescription and/oradministration of care to address the dental conditions. Examples oftreatments to dental conditions include prescription and/oradministration of brackets/wires, clear aligners, and/or otherappliances to address orthodontic conditions, prescription and/oradministration of restorative elements to address bring dentition tofunctional and/or aesthetic requirements, etc.

The treatment planning system 1908 may include a computing devicecapable of reading computer-executable instructions. The treatmentplanning system 1908 may provide to a user (e.g., a user of thetreatment professional system 1906) software (e.g., one or morewebpages, standalone applications (e.g., dedicated treatment planningand/or treatment visualization applications), mobile applications, etc.)that allows the user to interact with subjects (e.g., those people whoseintraoral cavities are being imaged by the scanner/camera 1904),create/modify/manage treatment plans. The treatment planning system 1908may be configured to process 2D images captured at the scanner/camera1904, generate 3D dental models using the 2D images, and/or generatetreatment plans for subjects whose dentition has been scanned/imaged. Insome implementations, the treatment planning system 1908 identifies aninitial position of a subject's dentition, an intended final position ofthe subject's dentition, and/or a plurality of intermediate positions tomove the subject's dentition toward the intended final positions. Insome implementations, the treatment planning system 1908 operates withuser input, e.g., with a technician and/or dental professional managinga treatment plan. In various implementations, however, some or all ofthe modules of the treatment planning system 1908 can operate usingautomated agents and without human intervention.

In the example of FIG. 19, the treatment planning system 1910 includes a3D oral cavity modeling system 1910. The 3D oral cavity modeling system1910 may execute one or more automated agents that operate to process 3Ddental models of a subject's oral cavity using data from dental scansand/or dental images. Examples of data from dental scans/images that canbe used for this purpose include height map data, data about colorinputs and/or textures, data about scan/image qualities, data about howmany counts of raw scans are in height map pixels, etc. The 3D oralcavity modeling system 1910 may use data from dental scans and/or dentalimages to predict dental classes used to segment a 3D model of asubject's dentition. The 3D oral cavity modeling system 1910 may processthe 2D images using manual, semi-manual, or automatic processingtechniques. As will be described in greater detail herein, in somevariations the processing may be driven, performed and/or guided by amachine learning agent. The machine learning agent may be trained on avariety of different datasets and may be adaptively trained, so that itmay update/modify its behavior over time. Any machine learning agentherein may use a “classifier,” which as used herein, may include one ormore automated agents operative to learn how to assign classes to one ormore items and assign those classes to those items. A classifier may betrained by training data, which as used herein, may include any data toset initial observations from a classifier may learn and/or adapt.

“Segmenting” a representation of a subject's dentition, such as a 2Dimage or a 3D model of a subject's dentition, as used herein, mayinclude labeling dental classes in the representation. Non-exclusiveexamples of dental classes include items corresponding to a subject'soral anatomy (teeth, gums, lips, tongue, other oral soft tissue, etc.)and items not corresponding to a subject's oral anatomy (non-oralanatomical items (e.g., fingers), non-anatomical items (dentalappliances, foreign objects, etc.). Additional non-exclusive examples ofdental classes include: teeth and/or particular teeth (e.g., teethidentified by tooth shape and/or anatomical tooth number), gingiva, andother items (excess materials, e.g., the subject's palate, the subject'stongue, other oral soft tissue of the subject, a finger or othernon-oral part of the subject's body, a dental appliance on the patient'steeth, etc.). Segmentation may involve assigning each point in a 3Dmodel of a subject's dentition an appropriate dental class. In someimplementations, segmenting a 3D model of a subject's dentition mayinvolve determining whether the various region of the 3D modelcorrespond to specific teeth, gums, or excess materials, and labelingthose regions appropriately.

The 3D oral cavity modeling system 1910 may further use data from dentalscans and/or dental images to modify and/or update 3D models of asubject's dentition so they are more accurate and relevant to treatmentplanning. As examples, the 3D oral cavity modeling system 1910 maymodify interproximal regions, gingival boundaries, and/or other areas ofa 3D model to make these regions more accurate and/or truer depictionsto a subject's intraoral cavity.

In some embodiments, the 3D oral cavity modeling system 1910 executesautomated agents that use artificial intelligence and/or machinelearning to predict dental classes in 3D dental models using data fromdental scans and/or dental images. In some implementations, the 3D oralcavity modeling system 1910 uses a neural network to classify data fromdental scans and/or dental images into appropriate dental classes. As anexample, the 3D oral cavity modeling system 1910 may map height map dataand pixel data to a set of human-labeled segmented images. The 3D oralcavity modeling system 1910 may derive one or more processes that, whenexecuted, predict dental class labels directly from height maps. Invarious implementations, labels can be binary and/or discrete (e.g. withvalues corresponding to different dental classes), continuous (e.g.,values ranging through a target height map), etc. The 3D oral cavitymodeling system 1910 may use a conditional Generative AdversarialNetwork (cGAN) and/or any other machine learning system to classify datafrom dental scans and/or dental images into dental classes. As notedherein, the 3D oral cavity modeling system 1910 may be trained with alibrary of labeled and/or accurately modeled 2D dental scans and/ordental images.

The 3D oral cavity modeling system 1910 may process 2D dental scansand/or 2D dental images in one or more ways, including segmenting theimages, and/or enhancing the images, including the interproximalregions, the height maps, etc.

The 3D oral cavity modeling system 1910 may modify, e.g., segment, a 3Dmodel of a subject's oral cavity with modules or engines that mayperform operations using one or more processors for digitally processingthe 3D model, and in particular for processing 2D images associated withthe 3D model so that modifications made to the 2D images may betranslated, including mapped, to the 3D model. The 3D oral cavitymodeling system 1910 may be configured to receive data, such as subjectscan data and/or 3D model data either directly (e.g., from thescanner/camera 1904), and/or indirectly, such as from a memory storing adigital model and/or 2D scan images from the subject's oral cavity(e.g., on the treatment planning system 1908). The 3D oral cavitymodeling system 1910 may processes these images and/or 3D model(s) andmay output, including displaying, storing and/or transmitting, the 3Dmodel of the subject's oral cavity. In some variation the apparatus maybe part of another apparatus (e.g., system) for treating a subject,including for generating a treatment plan and/or generating a series ofdental appliances for performing the treatment plan.

The 3D oral cavity modeling system 1910 may generally improve the 2Dimages and/or the 3D models of the subject's oral cavity, which may beused in a variety of beneficial ways; in particular a segmented and/orcorrected 3D model as described herein may be used to generate atreatment plan for modifying (e.g., correcting) a subject's dentition.In any of the methods and apparatuses described herein, the 3D model maybe used to generate one or more (e.g., a series) of dental appliances,such as but not limited to orthodontic aligners for re-aligning teeth.As will be described in greater detail below, there are a number ofindication, treatments, and processes that may benefit from thesegmented and/or corrected 3D models and 2 images described herein. Thusany of the methods and apparatuses described herein may be part of amethod or apparatus (e.g., system) for performing any of thesetreatments, processes, or the like.

The 3D oral cavity modeling system 1910 may execute automated agentsthat use projection values of 2D dental scans and/or dental images toproject attributes of the scans/images onto a 3D model. As noted herein,pixel values within 2D dental scans and/or dental images may includeheight map information representing distances of objects from thescanner/camera 1904 or visible light information as observed at thelocation of the scanner/camera 1904. When projected to a 3D model of anobject (e.g., a subject's dentition) within an area of interest, theheight map information may represent depictions of corresponding facesof a mesh of the 3D model. Automated agents executed by the 3D oralcavity modeling system 1910 may further resolve one or more conflictsbetween providing segmentation results on 2D images. Conflicts can beresolved statistically and, e.g., can involve taking consensuses,determining probabilities that a specific segmentation result is valid,etc. As an example, the 3D oral cavity modeling system 1910 may executeagents that implement Bayesian rules to combine multiple segmentationresults with each other.

The 3D oral cavity modeling engine 1910 may use processed 2D images tomodify a 3D model, either by revising the 3D model (e.g., surfaces)based on the processed 2D images, and/or by mapping components in theprocessed 2D images to components in the 3D model. In some variations,the 3D model may be a mesh model of at least a portion of the subject'soral cavity, and may include mesh points. Individual or groups of meshpoints may include data that indicates features (labels, such as toothnumber, color, etc.) extracted from the 2D images, and/or from theprocessed 2D images.

The 3D oral cavity modeling system 1910 can accurately create and/orupdate a 3D dental model and the ability to predict multiple dentalclasses concurrently. The 3D oral cavity modeling system 1910 can alsoaccurately segment 3D models of an oral cavity and associated structures(e.g., teeth, gingiva and/or palatal region), where each point in the 3Dmodel, e.g., in some variations in a mesh forming the 3D model, that arelabeled according to an appropriate dental class.

FIG. 2 shows an example of a 3D oral cavity modeling system 250. The 3Doral cavity modeling system 250 may correspond to an example of the 3Doral cavity modeling system 1910, discussed further herein. It is notedthat, as an example of the 3D oral cavity modeling system 1910, the 3Doral cavity modeling system 250 may include modules and/or implementfunctionalities different than the 3D oral cavity modeling system 1910The 3D oral cavity modeling system 250 may include one or more enginesand datastores. A computer system can be implemented as an engine, aspart of an engine or through multiple engines. As used herein, an engineincludes one or more processors or a portion thereof. A portion of oneor more processors can include some portion of hardware less than all ofthe hardware comprising any given one or more processors, such as asubset of registers, the portion of the processor dedicated to one ormore threads of a multi-threaded processor, a time slice during whichthe processor is wholly or partially dedicated to carrying out part ofthe engine's functionality, or the like. As such, a first engine and asecond engine can have one or more dedicated processors or a firstengine and a second engine can share one or more processors with oneanother or other engines. Depending upon implementation-specific orother considerations, an engine can be centralized or its functionalitydistributed. An engine can include hardware, firmware, or softwareembodied in a computer-readable medium for execution by the processor.The processor transforms data into new data using implemented datastructures and methods, such as is described with reference to thefigures herein.

The engines described herein, or the engines through which the systemsand devices described herein can be implemented, can be cloud-basedengines. As used herein, a cloud-based engine is an engine that can runapplications and/or functionalities using a cloud-based computingsystem. All or portions of the applications and/or functionalities canbe distributed across multiple computing devices, and need not berestricted to only one computing device. In some embodiments, thecloud-based engines can execute functionalities and/or modules thatend-users access through a web browser or container application withouthaving the functionalities and/or modules installed locally on theend-users' computing devices.

As used herein, datastores are intended to include repositories havingany applicable organization of data, including images, 3D models,tables, comma-separated values (CSV) files, traditional databases (e.g.,SQL), or other applicable known or convenient organizational formats.Datastores can be implemented, for example, as software embodied in aphysical computer-readable medium on a specific-purpose machine, infirmware, in hardware, in a combination thereof, or in an applicableknown or convenient device or system. Datastore-associated components,such as database interfaces, can be considered “part of” a datastore,part of some other system component, or a combination thereof, thoughthe physical location and other characteristics of datastore-associatedcomponents is not critical for an understanding of the techniquesdescribed herein.

Datastores can include data structures. As used herein, a data structureis associated with a particular way of storing and organizing data in acomputer so that it can be used efficiently within a given context. Datastructures are generally based on the ability of a computer to fetch andstore data at any place in its memory, specified by an address, a bitstring that can be itself stored in memory and manipulated by theprogram. Thus, some data structures are based on computing the addressesof data items with arithmetic operations; while other data structuresare based on storing addresses of data items within the structureitself. Many data structures use both principles, sometimes combined innon-trivial ways. The implementation of a data structure usually entailswriting a set of procedures that create and manipulate instances of thatstructure. The datastores, described herein, can be cloud-baseddatastores. A cloud-based datastore is a datastore that is compatiblewith cloud-based computing systems and engines.

The 3D oral cavity modeling system 250 may include a computer-readablemedium and one or more processors (or may be configured for operatingone or more processors). In FIG. 2A the schematic of the 3D oral cavitymodeling system 250 includes a scan data collector engine 252, a toothnumbering engine 258, a 2D image identification engine 254, a 2D imageprocessing engine 256, a component construction engine 260, a 2D/3Dprojection engine 262, a 2D image datastore and a 3D projectiondatastore 266. The various engines (e.g., modules) and datastores of thesystem may be coupled to one another (e.g., through the examplecouplings shown in FIG. 2A) or to components not explicitly shown inFIG. 2A. The computer-readable medium may include any computer-readablemedium, including without limitation a bus, a wired network, a wirelessnetwork, or some combination thereof.

The engine(s) included in the systems, and in particular the 2D imageprocessing engine 256, may implement one or more automated agents (e.g.,artificial intelligence and/or machine learning agents) that process 2Dimages, as will be described in greater detail below. For example, invarious implementations, a 2D image processing engine 256 may implementone or more automated agents configured to determine segmentation basedon the 2D image or a collection of images. The automated agent may betrained using a prepared dataset, e.g., from within the 2D imagedatastore 264 that may be manually segmented. In some variations, anautomated agent may identify interproximal spacing in the 2D images, andmay be trained on a prepared dataset of 2D images.

The system shown in FIG. 2A may also include a tooth numbering engine258 that may use one or more techniques (including machine learningtechniques) to estimate an order of teeth based, e.g., on locationand/or a height map from various 2D images. The 2D image processingengine 256 and/or the 2D image identification engine 254 may receiveinput from the tooth numbering engine 258. In some variations theidentification of tooth numbering by the tooth numbering engine may beiterative, as the identified tooth numbering may be modified by thesystem.

As mentioned, the tooth numbering engine 258 may be automatically orsemi-automatically determine or suggest the numbering of the teethwithin the oral cavity. A tooth type identifier datastore may beconfigured to store one or more tooth type identifiers of differenttooth types. In some implementations, the tooth type identifierscorrespond to numbers of a Universal Tooth Numbering System, characterstrings to identify tooth types by anatomy, images or portions thereofto identify tooth types by geometry and/or other characteristics, etc.The tooth numbering engine 258 may implement one or more automatedagents configured to gather tooth type identifiers. In someimplementations, a tooth type identifier gathering engine may gather aseries of tooth type identifiers corresponding to the teeth in a humanbeing's permanent/adult dentition. The tooth type identifier gatheringengine may gather from a tooth type identifier datastore includinguniversal or other tooth numbering system, character identifiers,image(s), etc. corresponding to a person's adult teeth. In variousimplementations, the tooth type identifier gathering engine may providetooth types to other modules, as mentioned above.

In FIG. 2A, the 2D image identification engine 254 may identify which 2Dimages, including in some variations which projections from the 3Dmodel, may be processed to improve the 3D model, including fordetermining segmentation. These 2D images may be selected and/orgenerated and may be processed (e.g., using the 2D image processingengine) and may be stored in a processed 2D image database or Datastore(not shown in FIG. 2A).

The system of FIG. 2A may include a component construction engine thatmay apply the processed 2D images to the 3D model in order to refine the3D model. In some variations the system may reconstruct the 3D modelfrom the processed 2D images. In some variations the 3D model may bemodified, e.g., by segmenting components and/or reconstructingcomponents using the processed 2D images. For example, the processed 2Dimages may include segmentation information that may be used to label,e.g., mesh points or facets on the 3D model. In some variationsindividual component portion of the subject's oral cavity (e.g., theteeth, individual teeth, the gingiva, etc.) may be reconstructed fromthe processed 2D images, e.g., using the 2D/3D projection engine 262.

FIG. 2B shows an example of a schematic of a 2D image processing engine256. In this example the 2D image processing engine 256 may beconfigured to identify and correct spacing between oral cavitycomponents such as teeth, e.g., interproximal spacing. In othervariations the 2D image processing engine may identify and/or correctother features from the oral cavity components, such as segmentation,gingival/tooth edges, etc. In FIG. 2B, the 2D image processing engineincludes an interproximal separation engine 268 that may processidentified 2D images (e.g., received by the 2D image processing enginefrom the 2D image projection/identification engine 254) to identifyinterproximal regions between individual teeth, including identifyingplanes between the teeth. These planes may be used to segment the teeth,e.g., by a segmentation engine 272. Alternatively or additionally, theseplanes may be used to enhance the 2D images to determine the separationbetween the dental components, e.g., using a 2D image enhancement engine270 to enhance the 3D model.

FIG. 2C illustrates one example of a component construction engine 262that may receive the processed 2D images. In some variations thecomponent construction image may identify those (processed and/orunprocessed) 2D images that include a particular component, such as aparticular tooth, e.g., using a 2D component finding engine 276. Onceidentified, the images of the components may be merged, using aComponent 2D merging engine 278 to form one or more 3D component models.For example, the individual tooth components may be reconstructed fromthe processed 2D images in this way. In FIG. 2C, a gingival modelingengine 280 may also be included that may use the reconstructed toothcomponents to generate a separate gingival model, e.g., by subtractingthe reconstructed teeth from the original 3D model and using a 3Dprocessing engine, such as a hole-filling engine 282 to smoothly fillholes in the mesh of the gingival model. These 3D component parts maythen be combined into a single, segmented, 3D model of the oral cavity.This model may be used as described below, e.g., to track tooth movementduring treatment, to plan a treatment, to adjust a treatment, and/or tomake or form an orthodontic appliance such as an aligner.

For example, a 3D model of the subject's oral cavity (e.g., dentition,gums, etc.) may be used to fabricate a dental appliance or a series ofdental appliances. In some variations an apparatus such as thosedescribed herein be part of or may include an aligner fabrication engine(not shown). An aligner fabrication engine(s) may implement one or moreautomated agents configured to fabricate an aligner. Examples of analigner are described in detail in U.S. Pat. No. 5,975,893, and inpublished PCT application WO 98/58596, which is herein incorporated byreference for all purposes. Systems of dental appliances employingtechnology described in U.S. Pat. No. 5,975,893 are commerciallyavailable from Align Technology, Inc., Santa Clara, Calif., under thetradename, Invisalign System. Throughout the description herein, the useof the terms “orthodontic aligner”, “aligner”, or “dental aligner” issynonymous with the use of the terms “appliance” and “dental appliance”in terms of dental applications. For purposes of clarity, embodimentsare hereinafter described within the context of the use and applicationof appliances, and more specifically “dental appliances.” The alignerfabrication engine(s) may be part of 3D printing systems, thermoformingsystems, or some combination thereof.

In use, a system such as illustrated above may be used to modify orimprove a 3D model of a subject's oral cavity. FIG. 2D illustrate oneexample of a method of modifying a 3D model as described herein. In thisexample, the 3D model is modified by segmenting (e.g., providingsegmentation values) the component parts. Similar steps may be used tomodify a 3D by improving the quality of the 3D model.

In FIG. 2D, the method of segmenting a 3D model of a subject's oralcavity (e.g., into individual teeth, gingiva, etc.) may being bycollecting a 3D model of the subject's oral cavity, which may includesome or all of the teeth, gingiva, palate, tongue, etc. This 3D modelmay be referred to as an “original” or “unprocessed” 3D model; thisoriginal/unprocessed 3D model may be modified as described herein, andthe modified 3D model may be referred to as a modified 3D model.Optionally, the method (and any apparatus configured to perform themethod) may include collecting 2D scans of the subject's oral cavity.The scans may be correlated to the 3D model; in some variations thescans may be used to form the original 3D model. For example, anintraoral scanner may be used to scan the subject's oral cavity andgenerate the 3D model of the subject's teeth. As described, in any ofthe methods and apparatuses herein, the 2D images (e.g., scans) may bemodified, e.g., by correcting, marking, etc., and these processed scansmay be used to regenerate the 3D model and/or in some variations theprocessing done in the 2D scans, such as segmentation, may be mapped tothe 3D model. Thus, as shown in FIG. 2D, a method may optionally includecollecting scans (2D images) of a subject's dentition and/or 3D model ofsubject's oral cavity (e.g., teeth, gingiva, etc.) 203. In somevariations collecting the 2D scans may include generating 2D sections,which may be taken through the 3D model. Although in FIG. 2D this stepis shown as an initial step, alternatively or additionally, this stepmay be performed later, such as after an initial analysis of the 3Dmodel (e.g., to identify tooth numbering, to identify occlusal lines,prior to segmenting, etc.).

Collected 2D images may then be analyzed to identify a subset of 2Dimages that include one or more features of the oral cavity to beprocessed, such as the teeth, gingiva, etc. 205. The identified 2Dimages, as mentioned, may be either or both scanned images and/orreconstructed images from the 3D model. The subset of images may beselected for inclusion into the subset based on a review of the contentof the images, to determine if the one or more corresponding featuresare present in the images. For example, individual teeth may beseparately and/or sequentially or iteratively examined and subsets ofthese images may be formed that include the tooth being examined at aparticular time. The subset may include a minimum and/or maximum numberof 2D images. In some variations a machine learning agent may be used toidentify the one or more features from the 2D images. In some variationsthe teeth in the 3D model and/or 2D images may be pre-processed, forexample, to number the teeth according to a standard dental numberingsystem. This preprocessing, such as numbering may be used to helpquickly identify which 2D images have the selected feature(s). A methodfor determining tooth numbering may also include machine learning, inwhich the machine learning agent (e.g., a tooth numbering engine) may betrained to identify tooth number, as discussed above.

As mentioned, in some variations some or all of the 2D images may begenerated as virtual sections through the original 3D model (or amodified version of the 3D model). The virtual sections may be taken soas to illustrate the one or more features.

The subset of 2D images may be processed 207. In some variations,processing may include (optionally) modifying the 2D images 209, such asindicating in some or all of the processed 2D images corrections to the2D images. For example, corrections may include determininginterproximal spacing, and/or correcting the interproximal spaces thatmay include, for example, scanning artifacts. Image processing mayinclude segmenting the 2D images 211. Segmentation may be performedusing a segmentation agent (or segmentation engine) that may apply oneor more rules to determine the boundaries of each tooth, such as theboundaries between the teeth and/or the boundaries between the teeth andthe gingiva, etc. In some variations the segmentation agent may be amachine-learning agent that is trained on one or more datasets torecognize boundaries between teeth or teeth and gingiva and to otherwisesegment the teeth and/or gingiva. Segmentation may be performed on the2D images and may be projected onto the 3D model (e.g., the original 3Dmodel or an intermediate 3D model that is modified). In general,corrections or modifications of the 2D images may be translated to the3D model, including by projecting onto the 3D model 213. For example,segmentation of teeth from the 2D images may be projected onto the 3Dmodel; when the 3D model includes a mesh structure having a plurality ofmesh points defining the structure, these mesh points may be labeled orotherwise marked to indicate that they are part of a particularstructure e.g., may be segmented). The mesh may be modified so that theindividual structures (e.g., teeth, gingiva, etc.) may be separate fromeach other (e.g., having separate mesh structures) that may share acommon reference frame. The segmented teeth and/or gingiva may then bemanipulated during later processing, such as when designing a treatmentplan and/or forming orthodontic appliances based on a treatment plan.

In some implementations, the hole filling engine 282 may be used with 2Dimages constructed from planes between teeth where the 2D imagerepresents the distance from the plane to the corresponding locations onthe 3D mesh. In this instantiation, the hole filling engine 282 can beused to reconstruct portions of the 3D mesh where no mesh preexisted.For example, to reconstruct the mesial and distal mesh edges of teeththat could not be reconstructed by the intraoral scanner.

In variations in which machine learning is used, for example, to performsegmentation of the 2D images, conditional Generative AdversarialNetwork (cGAN) and/or other neural network can be used. For example, insome variations a segmentation engine may include a machine learningagent to segment one or more 2D images, or image-like inputs, intovarious relevant dental classes. Many dental classes can be predictedconcurrently. Combining these predictions with knowledge of how the 2Dinputs project onto the 3D mesh may allow for improved 3D segmentation,as described herein. Thus, machine learning approaches can be used tosegment 2D inputs according to dental classes. The 2D machine learningpredictions can be projected to a 3D mesh to classify each point (e.g.,each point of the 3D mesh) and/or to modify the mesh. In somevariations, classification of each point can be achieved bystatistically combining the 2D images that support it (e.g., thatinclude the feature(s) that is/are being segmented).

For example, a 3D model may be formed using 2D images collected with anintraoral scanner. An intra oral scanner may work by moving the wandinside a subject's mouth to capture all viewpoints of every tooth.During scanning, the scanner may calculate distances to solid surfaces,e.g., from the wand (or the optics doing the scanning). These distancesmay be recorded as images called ‘height maps’. Each height map may beoverlapped algorithmically, or ‘stitched’, with the previous set ofheight maps to generate a growing 3D model. As such, each 2D image maybe associated with a rotation in space, or a projection, to how it fitsinto the 3D model. After scanning, the final model may include a set of3D points and their connections with each other (i.e. a mesh).

The apparatuses (including software) described herein may operate on themesh, and also on the 2D input images that are used to construct themesh, to (among other things) segment points into relevant dentalclasses, such as tooth, gingiva or moving tissue (tongue, fingers,etc.). As will be described in greater detail below, this labeled meshmay establish the basis of treatment planning for both orthodontic andrestorative cases.

Accurate mesh segmentation may be important for treatment planning. Asmentioned, the segmentation engine may use machine learning to segmentthe 2D images into their relevant dental classes described herein. Aconditional generative adversarial (cGAN) and/or other neural networkmay be used for segmentation, to learn how to map height map inputs (anexample of which is shown in FIG. 3A) to a set of human labeledsegmented images (e.g., FIG. 3B). The set can comprise several,hundreds, thousands, millions, etc. of human labeled segmented images.The result of this training may be a function that can predict labelsdirectly from height maps. Labels can be binary, many valued, with eachvalue corresponding to a different class, or they may be continuouslyvalued (such as a target height map). FIGS. 3A-3C illustrate identifyingteeth height maps from input height map images. FIG. 3C shows an exampleof a cGAN output.

Because the inputs may each be associated with a projection onto themesh, machine learning outputs can each be mapped to the appropriatepoints in the mesh. As such, each point in the mesh has support from oneor more 2D predictions. In some variations, conflicts in point labelsbetween the supporting 2D predictions can be resolved statistically,such as taking a consensus or using Bayes rule. An example of a raw 3Dmodel, projected to 2D from above is shown in FIG. 4A. In this example,the 3D model may be segmented by applying a segmentation engine thatuses machine learning to process the 2D images making up this mesh (or asubset thereof) and using resulting tooth segmentation predictions tolabel the 3D points (and remove non-tooth points); this is illustratedin FIG. 4B.

In some variations, points of the 3D model may be labeled (to indicatesegmentation and/or other corrections, modifications or results ofprocessing) when there are multiple 2D images using a technique such asBayes rule. For example, if there are calluses of only teeth (T) andother (O), the model may predict that each pixel is either positive (P)or negative (N) for teeth (or for a specific tooth number). Using a testset, one may calculate:

${P\left( T \middle| L \right)} = \frac{{P\left( L \middle| T \right)} \cdot {P(T)}}{\sum\limits_{c}^{T,N}{{P\left( L \middle| C \right)} \cdot {P(C)}}}$

A scanner, such as an intraoral scanner, may capture other informationin addition to height maps. This other information, corresponding to oneor more properties of the scanned structure (e.g., teeth, gingiva, etc.)may include, e.g., color inputs from a camera (textures), recordings ofscanning quality, and counts of how many raw scans contributing to eachheight map pixel. These inputs may be combined in any combination and/ormay be used jointly to predict dental classes by the segmentation engine(e.g., using machine learning) and/or may be included in the 3D model.An example of using these inputs is below. The inputs may include aheight map, such as is shown in FIG. 5A, a count map (an example ofwhich is shown in FIG. 5B), a grades map (an example of which is shownin FIG. 5C), and a texture for the image (an example of which is shownin FIG. 5D). FIG. 5E shows the target label (in this case ‘empty space’)and FIG. 5F shows a prediction from a machine learning engine trainedwith these inputs and targets, differentiating between empty andnon-empty space (black and white).

The methods and apparatuses described herein may be used, for example,to segment multiple labels at the same time (e.g., concurrently and/orsequentially). For example, machine learning outputs may be used topredict many labels simultaneously. In some variations, a differentoutput channel may be used for each label. For example, a three-channelRGB image may be generated with each dental label having a differentcolor. An example of this is shown in FIGS. 6A-6B. FIG. 6A shows aheight map input on the left, and the segmented image is shown in FIG.6B. In this example, the teeth are labeled as white 603, and excessmaterial (“non-teeth”) are shown labeled as grey (may be shown in acolor, such as red) 605. An entire subset of 2D images may be analyzedin this manner and the results may be combined to form a consensus(e.g., using Bayes rule or a comparable technique to distinguish betweenconflicting regions) that may be applied to the 3D model.

FIG. 7 illustrates one example of a method for segmenting a 3D model ofa subject's oral cavity (e.g., teeth, gingiva, etc.) as describedherein. Methods and apparatuses as described herein may be used tosegment a 3D model so that it may be separated, e.g., digitally, intovarious components. The components may be the individual teeth, thegums, implants, restorative tooth preparations, and other structureswithin the subject's oral cavity.

In the variation shown in FIG. 7, interproximal spacing, e.g., thespacing between teeth, may be used to identify the boundaries betweenteeth. For example, in some variations the apparatus may includeidentifying interproximals and calculating directions to view the 3Dmodel in order to optimally see the interproximal space. The views thatbest (e.g., maximally) show the interproximal spacing between two ormore teeth may be used to generate slices (e.g., 2D images, as describedabove) that may in turn be processed as described above; alternatively,the actual collected 2D images corresponding to these maximal views maybe identified from the set of scan images and used.

For example, 2D height map projections may be used, and these 2D heightmap projections may be improved, and these interproximal height mapprojection images may be improved to better-represent the interproximalregions. Thus, in some variations, the height map of differentcomponents shown in the 2D images may be used to segment the components,such as differentiating between a left tooth, right tooth, gingiva, airetc.

A selected component, such as a particular tooth, may be located in theimproved height map projection images and these 2D images may be refinedso to form improved height map projection images that include only theselected component. These improved projections may then be combinedusing a merge algorithm, such as marching cubes, to construct theselected component.

The procedure shown in FIG. 7 may be a special case of the methodillustrated and described above in FIG. 2D. In FIG. 7, a digital 3Dmodel of a subject's teeth may be received and and/or generated (e.g.,from a set of 2D images and/or height maps). In some variations the 3Dmodel and/or 2D images corresponding to the 3D model may be analyzed todetermine an initial tooth numbering (this initial tooth numbering maybe corrected or refined later) 701. For example, a top (e.g., occlusal)view of the teeth may be used to generate tooth numbering. The toothnumbering may be optional, but may be particularly helpful in the latersteps. FIGS. 8A-8B illustrate one example of tooth numbering. In FIG.8A, the top view, which may correspond to an actual top view, e.g.,taken from a single or composite scanned 2D image, and/or from aprojection of the 3D model) may be analyzed, e.g., by a tooth numberingengine, to determine the tooth numbering, as shown in FIG. 8B, in whichindividual teeth are numbered according to a standardized numberingscheme. FIG. 9 illustrates one example of a method of numbering theteeth, which may be automatic or semi-automatic. In FIG. 9, a height mapmay be used to first separate each tooth as a separate instance; toothnumbering may be calculated at the same time by calculating toothnumbering probabilities for each instance 901 (see, e.g., FIG. 10A). Forexample, jaw ordering may be assigned to all tooth instances, forexample, using a technique such as a Held-Karp algorithm for thetraveling salesman problem 903 (see, e.g., FIG. 10B). A maximumlikelihood estimate of all tooth probabilities jointly across the jawmay be determined, this maximum likelihood may preserve the dentalordering (e.g., molar->premolar->canine, etc.) 905. See, e.g., FIG. 10C.

Returning to FIG. 7, once the (optional) tooth numbering has beenperformed, and may be recorded in the 3D model or associated data aboutthe 3D model, the method may include iteratively identifying theinterproximal spacing between the teeth, including identifying a planethat maximizes the views of the interproximal region between the teeth.For example, the method (or an apparatus configured to perform it) maycalculate interproximal separation planes, e.g., the best planes thatseparate the relevant two teeth 703. An example of this is illustratedin FIGS. 11A-11B. In FIG. 11A, for every interproximal space found, themethod or apparatus may calculate separation planes, which are the“best” planes that separate the relevant two teeth. A planeperpendicular to these planes/lines 1104 may provide a view of the teeththat maximally shows the interproximal region. Thus, as shown in FIG. 7,the projections from buccal, lingual and occlusal views may bedetermined based on these interproximal planes/lines from the original3D model 705. As mentioned above, alternatively or additionally, actualscanned 2D images corresponding to the projections may be used.

The buccal, lingual and/or occlusal views identified as perpendicular tothe interproximal plane may be enhanced 707. In some variations machinelearning may be used to enhance the projections. For example, as shownin FIGS. 12A-12C illustrate buccal (FIG. 12A), lingual (FIG. 12B) andocclusal (FIG. 12C) projections through the original 3D model based onthe interproximal plane calculated from the 3D model. These views may beenhanced, as shown in FIGS. 13A-13B. FIG. 13A shows the same view as inFIG. 12A before enhancement of the interproximal region. FIG. 13B showsthe same view after enhancement of the interproximal region; as shown,the interproxial regions 1305 between the teeth have been enhanced andenlarged to correct for artifacts from the scanning. In FIG. 13B, thismay be done by trained network (e.g., using machine learning).

The same views, and/or additional views, may also be processed in otherways as well, including to determine the boundaries between thedifferent structures, for segmentation. For example, FIGS. 14A-14Billustrate the detection of segmentation boundaries in an enhanced image(FIG. 14A shows the enhanced image of FIG. 13B). In FIG. 14B thesegmentation engine, which may include a trained machine-learning agent,has detected different teeth, gums, dental scan bodies, etc. In thisexample, they are each indicated by a different color. Returning to FIG.7, the method may include segmenting the 2D images, e.g., by a trainednetwork, into different components, such as different teeth, gums,dental scan bodies, etc.) 709.

This process of calculating the interproximal planes, identifying 2Dimages perpendicular to the planes, enhancing these 2D images and/orsegmenting them may be repeated 711 until all of the interproximalplanes are identified and processed. All of the processed 2D images(e.g., projections) may be collected together.

Thereafter, individual components may be reconstructed from theprocessed 2D images. For example, the method may collect all of theprocessed 2D images that show a particular component 713, and may run amerging algorithm for this component (which may resolve conflictsbetween different images as described above, and may combine them into asingle reconstructed element 715. This is illustrated in FIGS. 15A-15F,showing a plurality of identified images including a particular elementor component, shown here as a tooth. These images may be merged into asingle representation of the particular element, as shown in FIG. 16.

The gingiva may be similarly reconstructed; in some variations, thegingiva may be segmented by subtracting the reconstructed teeth from therest of the 3D model, which is primarily the gingiva. As described inFIG. 7, the original scanned model may be modified by the reconstructedteeth from the earlier steps 719. For example, in the original 3D modelmesh representation of the 3D model may be modified by removing (e.g.,marking for deletion then deleting) all triangles or points that wereindicated to be part of one or more of the other elements from the 2Dimages, e.g., projections. Any holes or discontinuities that areidentified may be filled using a filling procedure to provide a finalsmooth gingiva that underlies the teeth 721. This is illustrated inFIGS. 17A-17B. FIG. 17A shows the 3D model with the segmented teeth;FIG. 17B shows the segmented model with the teeth removed, representingjust the gingiva. Finally, all of the segmented components may becombined, as shown in FIG. 18.

In general, these methods may allow for improving of interproximal spaceaccuracy, and for assisting detection of interproximal carries. Theimproved 3D images may also be useful for, in general, creating betterrendering and/or coloring of the tooth, e.g., by different materialreflection parameters to tooth and gum. As mentioned above, better 3Dmodels, and in particular, better segmented 3D models, may permit muchbetter treatment plan, and fabrication of more accurate orthodonticappliances, including better die separation.

For example, the methods and apparatuses described herein may allow theinput of just an initial 3D model, and may refine this model. Asmentioned above, 2D projections through the 3D model may be used forprocessing to improve the 3D model. Thus there may be no need forintermediate data from, e.g., a scanner.

In some variations, the output of the apparatus and methods describedherein when used to segment the 3D model may result in a 3D model thatis segmented into the composite parts, and may be readily separated intocomponent digital models of the different components. When interproximalspacing is used, as described in FIG. 7, above, the closed interproximalspaces on the 3D model may be opened up.

Thus, these methods and apparatuses may include multiple sources ofinformation that may be incorporated into model inputs and used jointlyfor prediction. Multiple dental classes can be predicted concurrentlyfor a single model or models. In addition, the accuracy may be higherthan traditional image and signal processing approaches.

FIG. 20 illustrates one example of a method as described herein. Thismethod may be performed by a processor, coupled with or in communicationwith a scanner (e.g., an intraoral scanner) or may otherwise receivescan data on a patient's dentition, or from a model of the patient'sdentition. In FIG. 20, the method includes accessing a plurality offirst two-dimensional (2D) images 2001, wherein the plurality of first2D images: represents a subject's oral cavity, each has first areas thatcan be segmented into a plurality of dental classes, each has a firstrelationship to a first three-dimensional (3D) model of the subject'soral cavity, and each has first height map data representing distancesbetween the subject's oral cavity and an image capture device. In somevariations, the method may also include accessing one or more automatedmachine learning agents trained to modify one or more second 3D modelsinto the plurality of dental classes 1003. The trained modifications mayuse second height map data of a plurality of second 2D images andfurther using second relationships between the plurality of second 2Dimages and the one or more second 3D models. The method may furtherinclude instructing the one or more automated machine learning agents touse the first height map data to modify the first areas of the pluralityof first 2D images to get a plurality of modified first 2D images 2005.In some variations the method may also include using the firstrelationships and the plurality of modified first 2D images to modifyfirst mesh regions of the first 3D model corresponding to the firstareas of the plurality of first 2D images 2007.

In some variations, the method may also include gathering the pluralityof second 2D images from a training datastore. The method may alsoinclude identifying one or more modifications to second areas of theplurality of second 2D images, and/or training the one or more automatedmachine learning to use the second height map data to provide the one ormore modifications to the second areas of the plurality of second 2Dimages to get a plurality of modified second 2D images. The method mayalso include training the one or more automated machine learning to usethe second relationships and the plurality of modified second 2D imagesto modify second mesh regions of the one or more second 3D modelscorresponding to the second areas.

These improvements in 3D model rendering and segmentation may thereforeprovide 3D shapes and 3D shapes with color that may improve the analysisof the subject's oral cavity and treatment planning. For example, theimproved 3D models resulting from the methods and apparatuses describedherein may provide a 3D shape and/or color that is sufficientlysegmented to allow more accurate modeling and formation of toothaccessories including artificial teeth, veneers, fillings, etc. The 3Dmodels described herein may include accurate colors, including scannedcolors and may improve the color properties, such as reflectivity, etc.The optical properties of the non-tooth components, such as gums andpalate may also be accurately rendered. Transparency, particularly forincisory teeth, may also be determined and/or modeled.

Any of the methods and apparatuses described herein may be used on acompleted scan, on a non-final scan, and/or while the teeth are activelybeing scanned. For example, in some variations, the methods andapparatuses described herein may be used to provide feedback to a userthat is scanning or that may go back to re-scan or continue scanning thesubject's teeth. Thus, these methods may indicate when there areincomplete or poorly-represented regions of the teeth, e.g., identifyingholes or gaps in teeth or between teeth and gums. For example, theapparatus or methods may include telling the user to complete a scan ofa particular region of the oral cavity (e.g., to re-scan tooth number13, etc.).

Additional advantages of these methods and apparatuses may includeimproving the 3D shapes, restorative treatments, and diagnostics. Forexample, dental and orthodontic treatments may be improved by knowingthe accurate identity and morphology of each tooth and the ability toprovide treatment to specific teeth, using information specific to eachtooth type. For example, these methods and apparatuses, and theresulting improved 3D models, may also allow for improve soft-tissuedetection and access material, including identifying the boundariesbetween teeth, which may also help improve inter-proximal spacing. Insome variations the teeth may be rescanned successive 3D models madeover time to more accurately track tooth movement, cavities, gumrecession, etc.

The methods and apparatuses described herein may also improverestorative treatments. For example, the improved 3D models, which mayinclude accurate color, reflectivity, and transparency of the teeth maybe used to show the effects of treatments such as tooth whitening,veneers, etc. in a more accurate manner. In some variations thesemethods may allow the teeth to identify incisors (e.g., showing veneertreatments), etc. The improved 3D models may also be used to helpdefine, display and examine treatments such as displaying crown shapes,etc. The accurate tooth numbering and modeling may also assist inautomatically generating and/or selecting treatment plans.

In addition, diagnostics may be improved by the methods and apparatusesdescribed herein. For example, these methods and apparatuses may beuseful to help with gum recession (e.g., gum recession diagnostics,including looking at longitudinal data, e.g., date over time), andgenerally looking at changes in the subject's oral cavity over time,including both global and regions changes. This may allow and supportimproved progress tracking, which may be part of a scanner (e.g.,intraoral scanner) system that may include this functionality. Thus,these methods and apparatuses may be used to diagnose tooth wear, andthe improved segmentation in particular may enhance the ability so seechanges in the teeth over time. In addition, the enhanced correlationwith the 2D images, and in particular the 2D images taken from the scandata may be use useful for following and measuring surface features onthe teeth such as plaque accumulation.

Any of the methods (including user interfaces) described herein may beimplemented as software, hardware or firmware, and may be described as anon-transitory computer-readable storage medium storing a set ofinstructions capable of being executed by a processor (e.g., computer,tablet, smartphone, etc.), that when executed by the processor causesthe processor to control perform any of the steps, including but notlimited to: displaying, communicating with the user, analyzing,modifying parameters (including timing, frequency, intensity, etc.),determining, alerting, or the like.

When a feature or element is herein referred to as being “on” anotherfeature or element, it can be directly on the other feature or elementor intervening features and/or elements may also be present. Incontrast, when a feature or element is referred to as being “directlyon” another feature or element, there are no intervening features orelements present. It will also be understood that, when a feature orelement is referred to as being “connected”, “attached” or “coupled” toanother feature or element, it can be directly connected, attached orcoupled to the other feature or element or intervening features orelements may be present. In contrast, when a feature or element isreferred to as being “directly connected”, “directly attached” or“directly coupled” to another feature or element, there are nointervening features or elements present. Although described or shownwith respect to one embodiment, the features and elements so describedor shown can apply to other embodiments. It will also be appreciated bythose of skill in the art that references to a structure or feature thatis disposed “adjacent” another feature may have portions that overlap orunderlie the adjacent feature.

Terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention.For example, as used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, steps, operations, elements, components, and/orgroups thereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items and may beabbreviated as “/”.

Spatially relative terms, such as “under”, “below”, “lower”, “over”,“upper” and the like, may be used herein for ease of description todescribe one element or feature's relationship to another element(s) orfeature(s) as illustrated in the figures. It will be understood that thespatially relative terms are intended to encompass differentorientations of the device in use or operation in addition to theorientation depicted in the figures. For example, if a device in thefigures is inverted, elements described as “under” or “beneath” otherelements or features would then be oriented “over” the other elements orfeatures. Thus, the exemplary term “under” can encompass both anorientation of over and under. The device may be otherwise oriented(rotated 90 degrees or at other orientations) and the spatially relativedescriptors used herein interpreted accordingly. Similarly, the terms“upwardly”, “downwardly”, “vertical”, “horizontal” and the like are usedherein for the purpose of explanation only unless specifically indicatedotherwise.

Although the terms “first” and “second” may be used herein to describevarious features/elements (including steps), these features/elementsshould not be limited by these terms, unless the context indicatesotherwise. These terms may be used to distinguish one feature/elementfrom another feature/element. Thus, a first feature/element discussedbelow could be termed a second feature/element, and similarly, a secondfeature/element discussed below could be termed a first feature/elementwithout departing from the teachings of the present invention.

Throughout this specification and the claims which follow, unless thecontext requires otherwise, the word “comprise”, and variations such as“comprises” and “comprising” means various components can be co-jointlyemployed in the methods and articles (e.g., compositions and apparatusesincluding device and methods). For example, the term “comprising” willbe understood to imply the inclusion of any stated elements or steps butnot the exclusion of any other elements or steps.

In general, any of the apparatuses and methods described herein shouldbe understood to be inclusive, but all or a sub-set of the componentsand/or steps may alternatively be exclusive, and may be expressed as“consisting of” or alternatively “consisting essentially of” the variouscomponents, steps, sub-components or sub-steps.

As used herein in the specification and claims, including as used in theexamples and unless otherwise expressly specified, all numbers may beread as if prefaced by the word “about” or “approximately,” even if theterm does not expressly appear. The phrase “about” or “approximately”may be used when describing magnitude and/or position to indicate thatthe value and/or position described is within a reasonable expectedrange of values and/or positions. For example, a numeric value may havea value that is +/−0.1% of the stated value (or range of values), +/−1%of the stated value (or range of values), +/−2% of the stated value (orrange of values), +/−5% of the stated value (or range of values), +/−10%of the stated value (or range of values), etc. Any numerical valuesgiven herein should also be understood to include about or approximatelythat value, unless the context indicates otherwise. For example, if thevalue “10” is disclosed, then “about 10” is also disclosed. Anynumerical range recited herein is intended to include all sub-rangessubsumed therein. It is also understood that when a value is disclosedthat “less than or equal to” the value, “greater than or equal to thevalue” and possible ranges between values are also disclosed, asappropriately understood by the skilled artisan. For example, if thevalue “X” is disclosed the “less than or equal to X” as well as “greaterthan or equal to X” (e.g., where X is a numerical value) is alsodisclosed. It is also understood that the throughout the application,data is provided in a number of different formats, and that this data,represents endpoints and starting points, and ranges for any combinationof the data points. For example, if a particular data point “10” and aparticular data point “15” are disclosed, it is understood that greaterthan, greater than or equal to, less than, less than or equal to, andequal to 10 and 15 are considered disclosed as well as between 10 and15. It is also understood that each unit between two particular unitsare also disclosed. For example, if 10 and 15 are disclosed, then 11,12, 13, and 14 are also disclosed.

Although various illustrative embodiments are described above, any of anumber of changes may be made to various embodiments without departingfrom the scope of the invention as described by the claims. For example,the order in which various described method steps are performed mayoften be changed in alternative embodiments, and in other alternativeembodiments one or more method steps may be skipped altogether. Optionalfeatures of various device and system embodiments may be included insome embodiments and not in others. Therefore, the foregoing descriptionis provided primarily for exemplary purposes and should not beinterpreted to limit the scope of the invention as it is set forth inthe claims.

The examples and illustrations included herein show, by way ofillustration and not of limitation, specific embodiments in which thesubject matter may be practiced. As mentioned, other embodiments may beutilized and derived there from, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. Such embodiments of the inventive subject matter maybe referred to herein individually or collectively by the term“invention” merely for convenience and without intending to voluntarilylimit the scope of this application to any single invention or inventiveconcept, if more than one is, in fact, disclosed. Thus, althoughspecific embodiments have been illustrated and described herein, anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

1. A computer-implemented method comprising: identifying a plurality oftwo-dimensional (2D) images of a subject's oral cavity, wherein theplurality of 2D images each comprise height map data corresponding to adistance to at least a portion of the subject's oral cavity, and theplurality of 2D images are each associated with one or more projectionvalues to relate the each 2D image to a digital three-dimensional (3D)model of the subject's oral cavity; processing the plurality of 2Dimages to segment each 2D image of the plurality of 2D images into aplurality of dental classes; and using the one or more projection valuesof each 2D image of the plurality of 2D images to project the segmented2D images onto the 3D model to segment the 3D model into a segmented 3Dmodel into the plurality of dental classes.
 2. The computer-implementedmethod of claim 1, further comprising collecting the plurality of 2Dimages and the height map data of each of the plurality of 2D imageswith an intraoral scanner.
 3. The computer-implemented method of claim1, further comprising collecting the plurality of 2D images byidentifying a view of the 3D model and generating a 2D projection of the3D model from the view.
 4. The computer-implemented method of claim 1,further comprising collecting the plurality of 2D images and the heightmap data of each of the plurality of 2D images from scanned images ofthe subject's oral cavity.
 5. The computer-implemented method of claim1, further comprising modifying the 2D images.
 6. Thecomputer-implemented method of claim 5, wherein modifying comprisesadjusting the height map data of each 2D image.
 7. Thecomputer-implemented method of claim 1, wherein processing the pluralityof 2D images comprises applying a trained machine-learning agent tosegment each of the 2D images.
 8. The computer-implemented method ofclaim 7, wherein processing comprises using a conditional GenerativeAdversarial Network.
 9. The computer-implemented method of claim 1,wherein projecting the segmented 2D images onto the 3D model comprisesresolving conflicts between the segmentation of each 2D image prior toprojecting onto the 3D model.
 10. The computer-implemented method ofclaim 9, wherein resolving the conflicts comprises applying Bayes'Theorem, deconflicting the conflicts through voting, or some combinationthereof.
 11. The computer-implemented method of claim 1, wherein the oneor more projection values of each of the plurality of 2D imagesrepresents a projection of pixels on the each of the plurality of 2Dimages to a face or vertex of a mesh of the 3D model.
 12. Thecomputer-implemented method of claim 1, wherein using the one or moreprojection values of each 2D image of the plurality of 2D images toproject the segmented 2D images onto the 3D model comprises mapping oneor more pixel values from pixels of the segmented 2D images onto one ormore faces of a mesh of the 3D model.
 13. The computer-implementedmethod of claim 1, wherein the plurality of dental classes compriseteeth, gums, and excess materials. 14.-21. (canceled)
 22. A systemcomprising: one or more processors; a memory coupled to the one or moreprocessors, the memory configured to store computer-programinstructions, that, when executed by the one or more processors, performa computer-implemented method comprising: identifying a plurality oftwo-dimensional (2D) images of a subject's oral cavity, wherein the 2Dimages correspond to a digital three-dimensional (3D) model of thesubject's oral cavity; processing the plurality of 2D images to segmenteach 2D image into a plurality of different structures; and projectingthe segmented 2D images onto the 3D model to form a segmented 3D model.23. The system of claim 22, wherein the computer-implemented methodfurther comprises: collecting the plurality of 2D images by identifyinga view of the 3D model and generating a 2D projection of the 3D modelfrom the view.
 24. The system of claim 22, wherein thecomputer-implemented method further comprises: collecting the pluralityof 2D images from scanned images of the subject's oral cavity.
 25. Thesystem of claim 22, wherein the computer-implemented method furthercomprises: modifying the 2D images.
 26. The system of claim 25, whereinthe computer-implemented method further comprises modifying the 2Dimages by adjusting a height map of each 2D image.
 27. The system ofclaim 22, wherein the computer-implemented method further comprises:processing the plurality of 2D images by applying a trainedmachine-learning agent to segment each of the 2D images.
 28. The systemof claim 22, wherein the computer-implemented method further comprises:processing using a conditional Generative Adversarial Network or otherNeural Network.
 29. The system of claim 22, wherein thecomputer-implemented method comprises projecting the segmented 2D imagesonto the 3D model by resolving conflicts between the segmentation ofeach 2D image prior to projecting onto the 3D model.
 30. The system ofclaim 29, wherein the computer-implemented method further comprises:resolving the conflicts by applying Bayes' Theorem. 31.-71. (canceled)